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SUMMARY 


Scores from standardized tests are common criteria for admission 
to a four-year college or university, and most require, recommend, 
or consider test scores as part of the admissions process. To help 
improve their chances of college admission, many high school 
students turn to test-preparation programs. We used an experimental 
design to evaluate the effectiveness of an online test-prep course to 
improve test scores. Specifically, given the current climate around 
test preparation and the call for more and better research evidence 
documenting the impact of test preparation, this research study 
sought to provide representative, relevant, and rigorous research on 
the use of ACT® Online Prep (AOP). 


SO WHAT? 


Overall, most students did not use the course as intended. Despite all 
the students in our sample expressing interest in using the product for 
free prior to their test date, many of the students who were assigned to 
the treatment group did not actually use the product during the month 
prior to their test date. These results indicate that simply granting 

a population of students access to a self-directed test preparation 
product such as AOP would not be sufficient for improving the college 
readiness of this population, as many of these students would not 
make adequate use of the product. Students’ may not be self-directed 
enough to engage with an unstructured test preparation program 
despite motivation to perform well on a standardized test. 


NOW WHAT? 


More work needs to be done within and external to the program to 
both encourage and create the necessary structure for students to 
become more consistently engaged with the program over a prolonged 
period in order to study its impact under these conditions. This study 
highlights some key avenues for future research on the ACT Online 
Prep product. We recommend qualitative research be conducted 

to better understand student motivational issues. Another issue 
surrounding usage of AOP is the lack of guidance on how to effectively 
utilize the program. To this end, we are currently working with schools 
that use AOP to learn about the dissemination and use strategies 

they are employing. Both of these types of studies, motivational and 
utilization strategies, will lay a solid foundation for understanding for 
whom, and under what conditions, AOP is most effective. 
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No Pain, No Gain: Lessons from a Test Prep Experiment 


Edgar I. Sanchez, PhD, and Ty M. Cruce, PhD 


Introduction 


Scores from standardized tests such as the ACT® 
or SAT® are common criteria for admission to a 
four-year college or university. According to 
recent data from the Integrated Postsecondary 
Education Data System (IPEDS), 88% of the 
nation’s four-year selective-admission institutions 
report that they require, recommend, or consider 
test scores as part of their admissions process 
(U.S. Department of Education, National Center 
for Education Statistics, Integrated 
Postsecondary Education Data System, 2016). 
These 1,555 colleges and universities that use 
test scores during the admission process enroll 
95% of all first-time students at four-year 
selective-admission institutions. 


There is a generally held belief that test- 
preparation programs can result in significant 
score gains and a postsecondary competitive 
edge (MacGowen, 1999; McDonough, 1997; 
Cole, 1987), and this belief drives many college- 
bound students to participate in these programs 
to improve their scores and thus their chances 
for college admission and scholarship 
opportunities. Since 1938 when Stanley H. 
Kaplan began tutoring students to prepare for 
the SAT, the phenomenon of test preparation 
has had a firm presence in American education. 
By 1981, when Adam Robinson and John 
Katzman founded The Princeton Review there 
was an appetite for these types of services. 
Fast-forwarding to the 2016-2017 academic 
year, 24% (903,038) of students who took an 
ACT test reported using some form of test 
preparation, and over 60,000 places of business 
in 2018 were estimated to offer exam 
preparation and tutoring (Barnes Reports, 2017). 
Although the test preparation industry in the 
US—currently worth an estimated $10 billion—is 
thriving as a result of this assumption, rigorous, 
research-based evidence in evaluation of this 


assumption has been challenging for 
researchers, and the test preparation efficacy 
claims based on this research have 

received increased scrutiny by the school 
counseling community. 


There are many studies that suggest that test 
preparation activities, including practice tests 
and retesting, have a small but notable 
association with score gains on standardized 
tests such as the ACT or SAT (e.g. Andrews & 
Ziomek, 1998; Cole, 1987; Powers & Rock, 
1998; Moss, 1995; Becker, 1990; Kulik, Bangert- 
Drowns, & Kulik, 1984; DerSimonian & Laird, 
1983; Scholes & Lain, 1997; Scholes & McCoy, 
1998). For example, a review of test preparation 
research by the National Association for College 
Admissions Counseling (Briggs, 2009) found 
test preparation solutions have minimal effects 
on test scores—an average gain of only around 
30 points on the SAT and less than one point on 
the English, reading, and math sections of the 
ACT, both of which are within the standard error 
of measurement for their tests. A potential key to 
understanding these effect sizes may be in 
understanding what is called “test preparation.” 
Broadly speaking, test preparation can 
encompass activities that mirror classroom 
instruction and focus on content skill 
development, or it can encompass test-taking 
skills (i.e., tips and tricks). In the former case, 
the test preparation is addressing core content 
knowledge, while in the latter case, test 
preparation utilizes strategies that are unrelated 
to content knowledge to improve test scores. 
Research has found that longer-term 
preparatory activities such as taking rigorous 
coursework have a greater association with 
score gains than shorter-term activities such as 
using workbooks, workshops, or learning test- 
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taking skills (ACT, 2005; Allensworth, Correa, & 
Ponisciak, 2008). 


According to a 2009 NACAC discussion paper 
addressing prior research in this area, most of the 
existing studies of the effects of test preparation 
rely on small, non-representative samples and 
suffer from methodological limitations that narrow 
their findings (Briggs, 2009). The discussion 
paper also notes that “the optimal design for an 
evaluation of the effect of test preparation would 
involve the random assignment of students into 
different preparatory conditions. To date no such 
study has been successfully conducted on a 
large scale” (p. 16). Finally, the discussion paper 
calls upon the research community to specifically 
document the magnitude of test preparation 
effects for the ACT, which the author felt was 
lacking in the literature. 


More recent studies confirm previous research 
findings of an association of modest score 
gains with test preparation for the ACT or SAT 
(i.e., typically within or near the standard error 
of measurement; Schiel & Valiga, 2014a; 
Schiel & Valiga, 2014b; Schiel & Valiga, 2014c; 
Appelrouth, Zabrucky, & Moore, 2015; 
Buchmann, Condron, & Roscigno, 2010; Moss, 
Chippendale, Mershon, & Carney, 2012; Lane 
et. al 2009).1 As has been noted, though, the 
benefits of test preparation, particularly very 
expensive test preparation, remains unclear 
(Adams, 2011). Despite the call by NACAC for 
the use of more rigorous research designs, most 
studies continue to struggle with relatively small 
samples and do not use more rigorous 
methods such as experimental or quasi- 
experimental designs. 


Given the current climate around test 
preparation and the call for more and better 
research evidence documenting the impact of 
test preparation, this research study sought to 
provide representative, relevant, and rigorous 


research on the use of ACT Online Prep (AOP), 
a self-directed online test preparation program 
currently with over 350,000 active accounts. To 
overcome the limitations of prior research in this 
area, we used randomized controlled trials to 
evaluate the short-term effectiveness of AOP in 
improving the ACT scores of students who had 
registered to take the test for a second time. 
This approach gives us a pre-test/post-test 
design with random assignment to treatment 
and control groups. 


Conceptual Framework 


AOP was developed for the purpose of 
improving students’ academic readiness for 
college-level coursework as measured by higher 
test scores on the ACT. To meet this purpose, 
AOP is hypothesized to improve several 
intermediary outcomes that have been 
empirically associated with improved test scores 
(see Figure 1). AOP familiarizes students with 
the content areas measured by (and the item 
types appearing on) the ACT and identifies 
those content areas where the student needs 
improvement. It then assists the student in 
skilling up in those identified areas through a 
personalized learning plan comprising 
instructional lessons and practice sessions 
along with tools to track progress and daily goals 
to help students stay on target. Prior research 
on the effects of test preparation and test 
familiarity have shown small positive gains in 
test scores. Several meta-analyses of studies of 
the effect of test preparation on test scores 
(Bangert-Drowns, Kulik, & Kulik, 1983; Powers, 
1993; Montgomery & Lily, 2012) have 
consistently found average test score gains of 
approximately one-quarter standard deviation. 
Similarly, a meta-analysis of practice effects 
(i.e., test content and item familiarity) by 
Hausknecht, Halpert, Di Paolo, and Gerrard 
(2007) found an average test score gain of 
about one-quarter standard deviation. 
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Figure 1. ACT Online Prep Conceptual Framework 
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Higher academic self-beliefs 


In addition to raising students’ levels of test 
familiarity and content mastery, AOP is 
hypothesized to increase academic self-beliefs 
and lower text anxiety, factors that have also 
been empirically linked to higher test scores. Ina 
meta-analysis of 55 studies, Valentine, DuBois, 
and Cooper (2010) found that students’ academic 
self-beliefs (i.e., self-concept, self-esteem, and 
self-efficacy) had a small positive incremental 
effect on academic achievement (i.e., grades or 
standardized test scores) after statistically 
controlling for a comparable prior measure of 
academic achievement. Hembree’s (1988) meta- 
analysis of 73 studies of the relationship between 
test anxiety and test performance found that 
students with low levels of test anxiety scored on 
average about one-half standard deviation higher 
on a standardized test than students with high 
levels of test anxiety (i.e., a stronger effect than 
that found for test preparation). 


To influence these intermediary outcomes, AOP 
was designed with different components such as 
practice tests and instructional lessons that 
contain an estimated 200 hours of content and 
activities that are aligned with the blueprints for 
the ACT subject tests. Specifically, AOP offers 
multiple short and long forms of ACT-produced 
practice tests in English, mathematics, reading, 
and science, and it offers predicted subject test 
scores and a predicted Composite score based 
on practice test results. Instructional lessons 


within AOP provide an in-depth review of the 
content covered by each section of the ACT test 
and are an opportunity for students to review 
domain content and improve content mastery. 
Finally, practice sessions provide students with 
an opportunity to take over 2,400 practice items 
covering all sections of the ACT. The games and 
flashcards provide students with an opportunity 
to test their knowledge of specific concepts 
covered by the ACT. Student’s experience with 
the different components within the program is 
largely self-directed. 


Study Design 


Minimizing Selection Effects 


In practice, selection into a paid test preparation 
program like AOP is not random, but rather is 
based on the students’ willingness to pay for the 
program and their willingness to use the 
program. As a result, observed differences in 
average test scores between those students 
who do and do not use AOP can be biased by 
self-selection. To address potential selection 
bias, we incorporated the following elements into 
our design. First, to minimize any potential 
selection bias that could be attributed to 
differences in the students’ willingness to pay for 
AOP, we received permission to award roughly 
2,500 AOP subscriptions for free to our study 
participants. Second, to minimize any potential 
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selection bias that could be attributed to 
differences in the students’ willingness to use 
the product, we emailed prospective study 
participants to gauge their interest in 
participating in a study in which they would be 
given AOP at no cost to prepare for their 
upcoming ACT test. Among those students who 
expressed an interest in using the product, we 
randomly assigned students to our treatment 
group (i.e., received free AOP) or a control 
group (i.e., did not receive free AOP). 


Treatment 


The treatment for this study was a free 
subscription to AOP during the five to six weeks 
between the registration deadline for an ACT 
national test date and the national test date itself. 
Although students can sign up for AOP at other 
times, the vast majority of students register for 
AOP at the same time that they register for an 
ACT test date, so the time frame in which AOP is 
used in our study is consistent with the time frame 
in which the program is often used in practice. Our 
goal in this research was to gauge the 
effectiveness of AOP not as it is intended to be 
used, but rather as it is actually used by students. 
As such, our treatment did not include any attempt 
by us to encourage more usage of the product or 
different usage patterns within the product. 


Sample and Study Protocol 


For this study, we conducted randomized 
controlled trials leading up to two different ACT 
national test dates: October 2016 and April 
2017. We chose two different test dates 
because we wanted a representative sample of 
both 11th graders and 12th graders in our study. 
Historically, registrations for fall test dates are 
predominantly 12th graders, whereas 
registrations for spring test dates are 
predominantly 11th graders. We pooled the 
samples together across the two trials for a 
single analysis of the effectiveness of AOP. 


For both trials, our target population comprised 
students who met the following criteria: (a) 
registered for the selected upcoming ACT 


national test date, (b) tested on a prior ACT 
national test date, (c) indicated on the test-day 
survey for the prior test date that they had not 
participated in any test preparation leading up to 
that prior test date, and (d) did not have a 
current or prior AOP subscription. Using these 
criteria, we pulled our target population from the 
ACT registration records that were available to 
us as of the registration deadline for the 
upcoming ACT test date. 


From this target population, we emailed a 
random sample of 87,718 students across the 
two test dates asking about their interest in 
participating in a study in which they would be 
given AOP at no cost to prepare for their 
upcoming ACT test (see Appendix A). As per the 
email instructions, those recipients who were 
interested in receiving AOP for free were 
required to express their interest by completing 
a short online survey (see Appendix B) that was 
accessible through a hyperlink embedded in the 
email. Of the 7,085 students who responded to 
the survey indicating that they were interested in 
participating in our study, we selected 5,106 for 
our study, randomly assigning 2,553 to the 
treatment group and 2,553 to the control group.” 
Members of the treatment group were notified by 
email that they were selected for the study and 
were provided with next steps for setting up their 
AOP account (see Appendix C). Members of the 
control group (and those not selected for our 
study) were notified by email that they were not 
selected for the study and were thanked for 
expressing their interest (see Appendix D). To 
ensure that treated students were given the 
most time possible with AOP, the identification 
of our target population and the recruitment and 
selection process were completed within one 
week of the registration deadline for the 
upcoming test. 
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Analysis 


We utilized an intention-to-treat (ITT) approach 
to our analysis, which requires that we include 
all students in the analysis based upon their 
initial assignment to either the treatment or 
control group. This approach is different than a 
per-protocol approach, whereby only those 
students who fully comply with the treatment are 
included in the analysis. We use the ITT 
approach because any decisions that students 
make after their random assignment (e.g., non- 
compliance with treatment, dropout, 
contamination, etc.) are not assumed to be at 
random and thus may bias the study results. 
Although ITT helps to guard against potential 
threats to internal validity, this approach 
changes the nature of our research question 
from the effectiveness of AOP as received to the 
effectiveness of AOP as assigned. Given this, 
an ITT approach will result in a more 
conservative estimate of the effectiveness of 
AOP. The trade-off for this more conservative 
estimate is that the finding will be unbiased and 
generalize back to the target population. 
Compared to a per-protocol analysis, ITT 
analysis is expected to provide a more realistic 
estimate of the effect of introducing AOP to a 
population of students because, in practice, not 
all members of that population would accept, 
comply with, or complete AOP. 


Using an ITT approach, we estimated five 
models regressing the post-test score in each 
ACT subject area (i.e., English, mathematics, 
reading, and science) and the post-test 
Composite score on the respective pre-test 
score, the students’ treatment status, and a set 
of covariates. Although random assignment 
created well-balanced groups (see Table 1), we 
elected to statistically control for additional 
covariates that we felt were important predictors 
of students’ post-test scores. Specifically, we 
included in our models the amount of time that 
had elapsed in months between the students’ 
pre- and post-test dates, as we felt that this was 
a necessary control given that not all students 
had taken the pre-test on the same test date and 
that academic growth that occurs during high 
school would differ by the amount of time that 
had elapsed between test dates. We also added 
an additional set of covariates for the students’ 
high school grade level, race/ethnicity, gender, 
and parents’ education level, as prior ACT 
research (Sanchez, 2013; Bassiri, 2016; 
McNeish, Radunzel, & Sanchez, 2016) has 
found variation in ACT test scores by these 
covariates in both cross-sectional and 
longitudinal models. Finally, we also included 
the trial (i.e., Fall and Spring) in which the 
students’ participated as a way to capture any 
unobserved differences in the students across 
the two trial dates. Descriptive statistics for the 
treatment and control group variables in our 
models are provided in Table 1. 
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Table 1. Descriptive Statistics for Treatment Group vs. Control Group, Full Sample vs. Post-test Sample 


Full Sample Sample with Post-test 
Treatment Control Treatment OTe) al iKe) | 
Percent/ Percent/ Percent/ Percent/ 
Variable Mean 1B) Mean 1B) Mean 1B] Mean 1B] 
African American 13.0 14.3 12.8 14.5 
Asian American 10.6 8.7 10.6 8.8 
Hispanic 9.8 10.4 9.9 10.0 
Other Race 6.1 5.5 5.8 O.0 
Unknown Race 8.4 9.2 8.4 9.3 
White O2Fl 51.9 By 15) 51.9 
Parent Ed: Unknown 12.6 13.4 12.4 13.0 
Parent Ed: No College 7.0 6.9 6.9 6.9 
Parent Ed: Some College 18.2 19.3 18.2 19.0 
Parent Ed: Bachelor's 
Degree 29.9 29.8 30.4 30.1 
Parent Ed: Graduate 
Degree 32.4 30.6 S2e2 31.0 
Female 62.4 61.2 62.6 61.9 
Male 37.6 38.8 37.4 38.1 
10th Grade 10.3 9.6 10.6 9.9 
11th Grade 39.6 39.1 39.8 39.3 
12th Grade 47.0 47.2 46.9 47.4 
Other Grade Sail 4.2 Ball 3.5 
Trial 1 65.1 65.1 64.7 64.7 
Trial 2 34.9 34.9 SS.6) 35.3 
Time Between Tests Tall 6.3 7.6 5.9 
Pre-test: English 22.9 6.1 22.7 6.1 23.0 6.1 22.8 6.0 
Pre-test: Mathematics 22eZ 5.1 22.1 5.0 Zee Sal 22:2 5.0 
Pre-test: Reading 2315 6.0 23.3 5.9 23.6 on 23.4 5.9 
Pre-test: Science 27258) 4.7 22.4 4.7 ZAG 4.6 2215 4.7 
Pre-test: Composite 22.9 49 22.7 4.8 FRAO) 4.8 22.8 48 
Post-test: English 24.8 6.2 24.6 6.2 
Post-test: Mathematics PAB 8) SZ 23.2 5.1 
Post-test: Reading 24.9 6.1 24.7 6.1 
Post-test: Science Zon Sal 23.4 5.1 
Post-test: Composite 24.3 5.1 24.1 51 


N 2,909 2,953 2,345 2,359 
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Limitations 


Although all participants in our study were 
registered for their respective national test dates, 
we discovered that not all students were in 
attendance on their test day. In particular, 402 
students in total were absent on their test day, 208 
from the treatment group and 194 from the control 
group. Table 1 also provides descriptive statistics 
for the reduced sample that sat for the post-test. 
As the information in the table illustrates, the 
relative balance between the treatment and control 
groups that was created by random assignment 
for the full sample still holds for these observable 
characteristics after removing those participants 
that did not sit for their post-test. These descriptive 
statistics suggest that the missing post-test data 
should not introduce bias in our results. Our 
models were therefore estimated only for those 
students who had post-test scores. 


Additionally, although all study participants 
indicated their interest in using AOP to prepare for 
their test, we discovered that a number of students 
who were randomly assigned to the treatment 
group did not follow through on their expressed 
interest and activate the AOP account that was 
provided to them. Specifically, 718 students 
across the two trials (28% of the treatment group) 
who were given complimentary access to AOP 
never activated their accounts.? Following our ITT 
analytic plan, these students were still included in 
the analysis. 


Results 


The parameter estimates and standard errors for 
the five models predicting post-test scores for the 
four subject tests and the Composite are 
presented in Table 2. As seen in the table, the 
coefficient for treatment group status was not 
statistically significant for any of the five models, 
meaning that the students who had short-term 
access to AOP at no cost did not perform 
differently than the control group on any of the 
subject tests or the Composite score. We believe 
the most likely reason why we did not find a 
meaningful group-mean difference associated with 
the treatment is that the usage rates within the 


AOP program were extremely low. Upon additional 
investigation, we found not only that 28% of the 
students who were randomly assigned to the 
treatment did not activate their AOP accounts, but 
also that an additional 27% activated their 
accounts but had zero active days within the 
program. This means that only 45% of the 
students who were randomly assigned to AOP 
were active within the program. Among the treated 
students who actually used the program, the 
average amount of time taking practice tests or 
practice sessions in the program was only 2.1 
hours. When we looked at student activity within 
AOP over time, we found that the cumulative time 
spent in the program increased exponentially over 
the days prior to the test date, suggesting that 
students were using AOP as a last-minute effort to 
improve their test scores. 


We were concerned that this low usage rate 
among students randomly assigned to the 
treatment was a function of some limitation in our 
study design; however, we discovered that this 
level of inactivity among our sample is actually 
consistent with the level of inactivity within the 
general population of students who purchase 
AOP. Of the almost 475,000 students who had 
purchased AOP as of the time of this writing, 51% 
had zero active days on the platform. Among 
those that had taken practice tests or practice 
sessions, the average amount of time engaged in 
the program was 8.3 hours. In this regard, while 
the percentage of students who had at least one 
active day in the system was consistent with the 
general AOP population, our experimental 
treatment group spent less time using the practice 
test and session sections than the general 
population of AOP users. We should note, 
however, that students in our study were granted 
access to the program for only one month, 
whereas the population of students who purchase 
AOP had access for up to 12 months depending 
on their particular subscription; this difference in 
access between our sample and the population of 
users may explain the difference in the time spent 
in the program. We also found a similar pattern of 
increased activity leading up to national test dates 
for this population of students as compared to our 
sample. 


ACT Research & Policy | ACT Research Report | R1757 8 


Table 2. Regression Results for ACT Post-test Score 


English Mathematics Reading Yor (1 a Tex) Composite 
Predictor 
Intercept 2.31 0.38 0.00 2.39 0.33 0.00 4.44 0.46 0.00 4.20 0.43 0.00 0.59 0.26 0.03 
Treatment Group 0.05 0.09 0.55 0.03 0.07 0.71 -0.06 0.10 0.59 0.10 0.09 0.27 0.04 0.06 0.47 
African American -0.83 0.14 0.00 -0.76 0.12 0.00 -1.27 Onli 0.00 -1.29 0:15 0.00 -0.61 0.09 0.00 
Asian American OZ 0.16 0.19 0.13 0.13 0.32 0.20 0.19 0.29 0.52 0.16 0.00 0.11 0.10 0.26 
Hispanic -0.09 0.16 0.59 -0.16 0.13 0.21 0.00 0.19 0.98 -0.26 0.16 0.11 -0.08 0.10 0.42 
Other Race -0.19 0.20 0.34 -0.04 0.16 0.78 -0.29 0.23 Ori -0.29 0.20 0.15 -0.17 Om 0.18 
Unknown Race 0.52 On 0.00 0.14 0.13 0.30 0.16 0.20 0.43 0.11 0.17 0.51 Om 0.11 0.10 
Parent Ed: 
Unknown -0.10 0.15 Foz 0.26 0.12 0.04 -0.01 0.18 0.97 0.13 0.16 0.42 0.11 0.10 0.25 
Parent Ed: No 
College -0.62 0.19 0.00 -0.45 0.15 0.00 -1.31 0.23 0.00 -0.84 0.20 0.00 -0.50 0.12 0.00 
Parent Ed: Some 
College -0.31 0.13 0.02 -0.36 0.11 0.00 -0.75 0.16 0.00 -0.51 0.13 0.00 -0.26 0.08 0.00 
Parent Ed: 
Bachelor's Degree -0.10 Oni 0.37 -0.10 0.09 0.27 -0.30 0.13 0.03 -0.24 0.12 0.04 -0.06 0.07 0.41 
Female -0.12 0.09 0.18 -0.40 0.07 0.00 -0.10 0.11 0.35 -0.38 0.09 0.00 -0.29 0.06 0.00 
10th Grade 2.34 0.29 0.00 2.38 0.24 0.00 72 ley 0.35 0.00 2.31 0.30 0.00 2.16 0.19 0.00 
11th Grade PIES) 0.28 0.00 1.96 O22 0.00 2.56 0.33 0.00 2.42 0.29 0.00 2.00 0.18 0.00 
12th Grade 1.42 0.28 0.00 1.18 0.23 0.00 sierra 0.34 0.00 1.38 0.29 0.00 1.28 0.18 0.00 
Time Between 
Tests 0.12 0.01 0.00 0.11 0.01 0.00 0.10 0.01 0.00 0.11 0.01 0.00 Om 0.01 0.00 
Trial 2 -0.09 On 0.44 -0.18 0.10 0.06 -0.62 Ons 0.00 -0.39 0.13 0.00 -0.13 0.08 0.09 
Pre-test Score 0.87 0.01 0.00 0.86 0.01 0.00 0.79 0.01 0.00 0.78 0.01 0.00 0.94 0.01 0.00 


N 4,684 4,685 4,682 4,678 4,678 
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Discussion 


The purpose of this study was to examine the 
effect of a particular test prep program—ACT 
Online Prep—on the test scores of students. An 
experimental research design was used to help 
overcome a number of the limitations of prior 
research in this area. In order to minimize any 
potential bias that would come from differences 
in students’ willingness to pay for and use AOP, 
we drew our study sample from a population of 
students who had expressed an interest in using 
the product for free over the month prior to their 
ACT test. We used a pre-test/post-test design 
with random assignment to the treatment and 
control groups to create balanced groups for 
comparison and used an ITT analytic approach 
to minimize any bias due to differences between 
groups that can occur after assignment (e.g., 
non-compliance, dropout, and contamination). 


Despite all the students in our sample expressing 
interest in using the product for free prior to their 
test date, many of the students who were 
assigned to the treatment group did not actually 
use the product during the month prior to their 
test date. Because of our use of ITT, these 
students remained in our treatment sample 
despite their non-compliance, and we found that 
none of the estimated differences in post-test 
scores between our treatment and control groups 
were Statistically significant. The proper 
interpretation of these null findings is that simply 
granting a population of students access to a self- 
directed test preparation product such as AOP is 
not sufficient for improving the college readiness 
of the population, as many of the students would 
not make adequate use of the product. 


While the present study did not find an effect for 
assignment to the AOP program, a finding that 
runs counter to much of the test preparation 
research, there are important differences to 
consider between our study and prior research. 
The most important issue is the manifest lack of 
participation of students in our treatment group. 
In our study, we utilized random assignment to 
treatment with an ITT methodology, which 
meant that students were retained in our 


treatment group regardless of their participation 
level. This methodology provides an unbiased 
estimate of the treatment effect but changes the 
study question from the impact of program use 
to the impact of program assignment. Prior 
studies that have focused only on active users of 
test preparation products may be addressing the 
question of program use, but their reliance on 
per-protocol analysis without random 
assignment suggests that estimates of the effect 
of test preparation on standardized test scores 
may be biased. 


There are also important methodological 
differences between our study and prior studies. 
As Briggs (2009) noted, relatively few studies 
have made use of random assignment or quasi- 
experimental designs. Notable exceptions 
include Domingue and Briggs (2009), Holmes 
and Keffer (1995), McMann (1994), and Scholes 
and Lain (1997). These studies found mixed 
results for the preparation programs examined. 
Of these studies which employed more rigorous 
research designs, Domingue and Briggs (2009) 
and Scholes and Lane (1997) found no 
statistically significant differences between those 
who used test preparation and those who did 
not. Interestingly, both of these studies relied on 
global indicators of test preparation use but did 
not follow up with more detailed information on 
student participation within the prep programs. 


The test preparation programs in these studies 
differ in, among other things, their samples, 
program examined, content covered, mode of 
delivery, and intensity of interaction with the 
programs. This makes comparisons across 
studies challenging. 


Like our study, these studies could benefit from 
an examination of the heterogeneity of program 
utilization. Furthermore, the studies that found 
an effect for test preparation examined single 
programs that were offered through their school 
as opposed to other studies that examined test 
preparation across environments including self- 
directed test preparation. 
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This study, while highlighting the more obvious 
“non-effect” associated with not utilizing the 
program as it is intended, has important 
implications for future practice. First, the results 
of this study not only shed light on the low usage 
rates of students, but they provide evidence that 
minimal use of AOP just before a test is not an 
effective approach to improving test scores. 
Decades of research into student learning finds 
that distributing practice over time is more 
beneficial for long-term retention than massing 
practice (Bjork & Bjork, 2014). Second, the 
results of this study suggest that students’ may 
not be self-directed enough to engage with an 
unstructured test preparation program despite 
expressing an interest in using the product to 
improve their performance on a standardized 
test. This reminds us of the work of Allensworth 
et al. (2008), who found that despite students 
expressing high motivation to do well on the 
ACT, 45% of them reported studying for the ACT 
on their own only “once in a while” or “never.” 
More work needs to be done within and external 
to the program to both encourage and create the 
necessary structure for students to become 
more consistently engaged with the program 
over a prolonged period in order to study its 
impact under these conditions. 


This study also highlights some key avenues for 
future research on the ACT Online Prep product. 
One of the key issues that impacted our study 
was that of motivation and utilization of the 
product. As almost 50% of our treatment group 
did not utilize the product, it is important to 
examine the motivational barriers to using test 
preparation. If we can understand when students 
elect to actively participate in test preparation, we 
can begin to get a better understanding of the 
efficacy of the program for students who use the 
program. We recommend qualitative research be 
conducted to better understand student 
motivational issues. Another issue surrounding 
usage of AOP is the lack of guidance on how to 
effectively utilize the program. To this end, we are 
currently working with schools that use AOP to 
learn about the dissemination and use strategies 
they are employing. Both of these types of 
studies, motivational and utilization strategies, 
will lay a solid foundation for understanding for 
whom, and under what conditions, AOP is 

most effective. 
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Notes 


1. The standard error of measurement estimates how repeated assessments by the same person on the 
same test would be distributed around a person’s “true” score. For example, the median SEM for ACT 
Composite score is 0.93. Therefore, we would expect a student to score within approximately one 
point if they were to take the ACT again without additional learning having taken place. The ACT SEM 
values are available at 
http://www.act.org/content/dam/act/unsecured/documents/ACT_Technical_Manual.pdf. 

2. Given this sample size, we had 87% power to detect an effect size of about one-half point on the ACT 
score scale. 

3. Less than 0.5% of our control group went on to purchase AOP during the study period. 
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Appendix A 


Example Invitation Letter 


Hello <First_Name>, 


Congratulations! You’ve taken a big step toward your future by registering to take the April 2017 ACT 
Test®. We know how important the ACT test is to you, and we want you to be successful! 


We are looking for high school students interested in participating in a test prep study this spring before 
taking the ACT in April. In this study we will give you free access to ACT Online Prep™, a $39.95 value, 
to use to prepare for your April ACT test. 

This study is the first of its kind from ACT, as we look for new ways to assist students in strengthening 
academic skills and preparing for the ACT. We look forward to learning how access to new test 
preparation materials can positively impact your preparation and performance on the ACT. 

By participating in this study we ask that you: 


e Activate your account on ACT Online Prep within 48 hours of being notified of selection 


e Make use of the resources that ACT Online Prep has to offer to help you prepare for your April 
ACT exam 


e Take the ACT test on April 8% 
e Complete a brief survey after you take the ACT sharing your experience using ACT Online Prep 
We look forward to the chance to help you prepare for your spring ACT test date! 


Are you interested in participating? Please follow the link below and complete the included two question 
survey to be considered for participation: <Survey_Link> 
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Appendix B 
Example Sign-up Survey 


Thank you for your interest in participating in our test prep study. This offer is exclusively for you. Please 
do not forward this offer to anyone else. To be considered for inclusion in the research study and be 
eligible to receive free access to ACT Online Prep you must complete this two question survey. 


| have read the terms and conditions. 


Which of the following test preparation activities, if any, are you planning to use to prepare for the April 8" 
ACT test? (Choose all that apply.) 


| will prepare for the ACT by myself 


| will prepare for the ACT with other people in a group setting 


| will prepare for the ACT with other people online in a group setting 


| will use 1-on-1 tutoring to prepare for the ACT test 


Other (please explain) 


| do not plan to do any other test preparation activities 


The following is a list of goals you may want to meet to prepare for the ACT test. Please rank the 
following goals from the most important to least important. Click and drag to rank items, where 1 is the 
most important, 6 the least. 


Reviewing what will be on the test 
Learning what to expect on the test day 
Learning about study habits and skills 
Learning about time management 
Learning test taking strategies 


Other (please explain) 


Congratulations Page Content <Final Page of the survey>: 


Thank you! You have registered your interest to participate in ACT’s test prep study. 


Be sure to check your inbox as we will be notifying those selected to participate in the next few days. That 
email will include log-in information and next steps. 


Slots for the study are limited so you need to confirm your account within 48 hours of receiving the 
selection email. If you do not, we will need to offer your slot to another student. 


Thanks for your interest! 
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Appendix C 


Example Acceptance Letter 


Hello <First_Name>, 


Congratulations! We are pleased to inform you that you have been selected to participate in the test 
preparation research study. As part of this study you will have free access to ACT Online Prep™, a 
$39.95 value, to use to prepare for your April ACT test. You will have access to the ACT Online Prep 
system through April 8th. 


For this study we ask that you make use of ACT Online Prep to help you prepare for your April test date 
and answer a short survey after your ACT test date. We encourage you to make frequent use of the 
resources that are available to you through this program. Each will help you take additional steps toward 
your goal of performing well on your April ACT test date. 


ACT Online Prep includes: 
e Mobile App — you will gain access to the free mobile app so you can prepare anytime, anywhere! 


e Quality learning content — Our library of lessons helps with reviewing content assessed on the 
ACT in all four subject tests. 


e Personalized learning path — You can get started with ACT Online Prep by taking a short-form 
ACT test to get a predicted score range. The system will create a learning path specific to your 
needs. 


e Guided plan — You can stay on track by entering your ACT test date; the learning path will adjust 
based on that timeframe. 


e Game center — You can choose from several different games and compete with other students to 
test your knowledge. 


e Tests — Our full-length practice test (which will give you a predicted ACT score) and practice 
questions are designed to simulate the actual exam, helping you get used to the content and 


format of the ACT so there will be no surprises on test day. 


You will be receiving a separate email with information about how to access the ACT Online Prep system 
very soon. 


Again, congratulations and best of luck on your April test date! 
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Appendix D 


Example Rejection Letter 


Hello <First_Name>, 


We thank you for your interest in participating in this research study on test preparation. Unfortunately 
you were not selected to participate in the study. We wish you good luck on your April ACT test. 


Thank you and good luck! 
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About ACT 


ACT is an independent, nonprofit organization that provides assessment, 
research, information, and program management services in the broad areas of 
education and workforce development. Each year, we serve millions of people 

in high schools, colleges, professional associations, businesses, and 
government agencies, nationally and internationally. Though designed to meet 

a wide array of needs, all ACT programs and services have one guiding 


purpose—helping people achieve education and workplace success. 


About ACT Research 


ACT Research leads the field with authority and high-quality scientific evidence 
in support of education and workforce practices, solutions, and services. Our 
mission-driven team comprises a variety of backgrounds and disciplines, and 
offers a wide spectrum of knowledge and skills, enabling us to deliver quality, 

high-impact products and services aligned to ACT’s strategy and mission. 
Together, our research teams provide policymakers, educators, parents, and 

learners with research-based insights to inform their decision-making, and 

deliver educators and workforce development professionals with tools and 


services needed for education and career navigation. 
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